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ABSTRACT 


In its 1996 report "Time for Business", the Small Business Deregulation Task 
Force recommended the establishment of a clearance process for 
Commonwealth Government business surveys as a means of ensuring that 
these surveys are necessary, well designed, and place minimum load on 
business respondents. Funding for survey clearance was allocated from July 
1997 onwards, and, in view of both its statistical expertise and its coordination 
role, the Australian Bureau of Statistics (ABS) was selected to conduct the 
process. In December 1997, clearance of surveys began. The entire suite of 
Commonwealth Government business surveys will be reviewed by June 
1999. Those that do not satisfy the review criteria are not permitted to start 
or, in the case of existing ongoing surveys, to continue. 


This paper outlines the genesis, objectives, development and operation of the 
Statistical Clearing House. Part A describes the objectives, challenges and 
framework within which developmental and operational activities are taking 
place. Part B focuses on the review criteria, procedures and systems, and 
the corresponding information about surveys that is required for the review 
and that is subsequently disseminated publicly via the Internet. Part C deals 
with the organisation of the Statistical Clearing House, including staffing, 
communications, coordination with other areas, and reporting and evaluation 
requirements. Part D summarises the findings of reviews conducted to date 
and the impact on respondent burden and quality. Part E raises the 
outstanding issues and outlines possible future directions. 


The primary question that we would like to put to Methodology Advisory 
Committee members is whether the survey review criteria and the information 
required to enable their application are appropriate. We would also 
appreciate suggestions on any other aspects of the clearance process and, in 
particular, how we can best market it to survey managers, consultants 
providing survey design and operations services, and the businesses 
themselves. 


PART A: BACKGROUND 
Genesis 


The Small Business Deregulation Task Force report proposed that "statistical 
collections affecting 50 or more businesses and run by, or on behalf of, all 
Commonwealth Government departments and agencies be subject to a 
central clearance process", that non-approved surveys not be permitted, and 
that approved surveys receive a registration number to be displayed on 
collection material. The report also outlined the review criteria it expected to 
see incorporated in the clearance process, and it further noted that clearance 
should not apply to statistics produced as by-products of administrative 
processes, that clearance should be administered by the ABS, and that, to 
ensure even handedness, all surveys, including those conducted by the ABS, 
should be cleared. 


The Commonwealth Government was fully supportive of the proposal in its 
March 1997 "More Time for Business" statement and allocated funding to the 
ABS for implementation. Thus, in July 1997, the Statistical Coordination 
Section, a self-contained unit within Methodology Division was established to 
develop and operate a "Statistical Clearing House" (SCH). 


To ensure that creation of the SCH was fully supported and publicised across 
all Commonwealth Government departments and agencies, a Cabinet 
Submission was prepared. It was discussed by the Cabinet in September, 
1997. The outcome was a letter from the Prime Minister to all Ministers 
stating that clearance of existing and proposed new surveys would begin on 1 
December 1997 and that the Australian Statistician would determine further 
details of the process and communicate them to the relevant agencies. 
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On 24 October, 1997 the Australian Statistician wrote to all departmental 
portfolio heads and to the heads of agencies known to have business survey 
activity. The letter and accompanying annex (copy appended) elaborated 
SCH procedures and requested that each department/agency nominate a 
"Survey Liaison Officer" to interface with the SCH. It also stated that 
information about approved surveys would be recorded in a "Commonwealth 
Register of Surveys" accessible via the Internet. 


Objectives and Benefits 


The SCH has three basic, interrelated objectives. The first is to reduce the 
load imposed by the Commonwealth Government on businesses, particularly 
small businesses, by eliminating duplication and by ensuring that the design 
and conduct of business surveys follow good practice. The second is to 
increase the value of survey outputs by improving the quality of the methods 
used. The third is to enhance the use of survey outputs by improving access 
to documentation describing them and the methods used to produce them. 


The SCH is expected to have a significant effect on respondent burden 
measurement and control. It is raising awareness of respondent burden 
amongst survey managers. In most agencies, respondent burden has never 
been monitored before, let alone managed. The SCH is requiring a 
respondent burden projection to be specified for each survey subject to 
review, followed by an estimate of the burden actually incurred. These data 
are recorded and will be disseminated through the Register of Surveys. They 
will enable the first assessment of the total burden imposed by 
Commonwealth statistical collections on business respondents, and the 
relative load imposed by each agency, including the ABS, as a proportion of 
the total. As the SCH stamp of approval on the front of survey forms 
becomes recognised over time, it will be viewed by business respondents as 
validating that a survey is necessary and well designed for its purpose. 


Not only does the SCH aim to reduce respondent burden and improve survey 
quality, it is also providing the ABS with an opportunity to enhance its 
statistical co-ordination role. Standards can be promoted through the SCH. 
The Register of Surveys will gradually increase the awareness and use of 
Commonwealth Government statistical data, and of the documentation 
describing these data and their production processes. 


Barriers and Challenges 


Notwithstanding its benefits, survey managers have a natural tendency to 
view the SCH, initially at least, as a bureaucratic impediment rather than as a 
potential source of improvements to survey procedures and outputs. Faced 
with tight deadlines and budgets, managers can be reluctant to set aside the 
time and effort to provide the information required for SCH review, or to make 
any changes required for their surveys to satisfy the review criteria. A major 
challenge for the SCH is to persuade survey managers that review by the 
SCH is more a potential source of value added than a policing operation. 


In fact, this will not be the first time that the ABS has engaged in a review of 
statistical collections. In 1983, there was a ministerial direction requiring 
Commonwealth agencies intending to conduct surveys to consult the ABS 
beforehand. This direction worked with diminishing success for two reasons. 
The ABS had only an advisory role and could not insist on changes to a 
survey. As well, the ABS could spare only limited resources for the review 
program. The process was finally abandoned in 1993 as being ineffectual. 
Thus, past experience indicates that there are some pitfalls to be avoided if 
the clearance process is to have the desired impact. 


The present SCH differs from the earlier approach in several ways that 
greatly increase its chances of proving effective. 

- The drive for a clearing process has come from outside the ABS. The 
SCH is being championed by the Office of Small Business on behalf of its 
constituency, with the primary aim of reducing provider load. 

- Funding has been allocated for SCH development and operation. 

- The clearing process is more narrowly focussed, including only surveys 
of 50 or more business respondents, rather than all surveys. 

- There have been significant improvements in the technology available 
to support the process. In particular, use of the Internet enables SCH 
information to be more readily accessed and updated. 


PART B: DEVELOPMENT 
Implementation Timetable and Strategy 


The implementation timetable has four phases. In the first phase, which 
lasted from program initiation until Christmas 1997, the focus was on 
recruiting staff, developing review and approval criteria, establishing the initial 
clearance procedures and systems, and informing stakeholders. 


The second phase, from December 1997 until June 1998, involves full scale 
clearance operations as well as continuing development. Reviews are being 
conducted of all new surveys prior to their going into the field and of the larger 
ongoing surveys. The review criteria are being refined in the light of 
experience and feedback from stakeholders. The Register of Surveys is 
being developed, and SCH procedures and systems are being enhanced. 


The third phase, from July 1998 until June 1999 will involve continuing review 
of all new surveys coupled with review of the remaining backlog of smaller 
ongoing surveys. As the review load diminishes, the SCH staff will increase 
its statistical coordination activities to include more active promotion of 
standards and harmonisation. 


Towards the end of 1998/99, there will be an independent evaluation of SCH 
development and operations. Preparations will be made to move the SCH 
into the fourth, steady state operational phase. SCH operations will be scaled 
down to a level sufficient to deal with the ongoing stream of new collections. 
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Strategic principles include: 

- making no distinction in the clearance process between ABS surveys 
and surveys conducted by other agencies; 

- allocating survey review effort in accordance with its likely benefits in 
terms of respondent burden reduction or survey improvement; 

- conducting clearance reviews in a timeframe that has minimum effect 
on the survey development timetable with, at the very worst, a turnaround 
time of 20 working days from the date complete information about a survey is 
received; 

- harmonising with and utilising any existing procedures and systems, 
within the ABS or outside, that perform related functions; 

- recording and disseminating survey review information electronically. 


Scope and Definitions 


As previously noted, the Task Force report proposed that "statistical 
collections affecting 50 or more businesses and run by, or on behalf of, all 
Commonwealth Government departments and agencies be subject to a 
central clearance process". The precise scope and definitions of the SCH 
were established within this framework and incorporated Paragraphs 1-27 of 
the Annex to the Australian Statistician's letter. 


"Statistical collections" are distinguished from "administrative collections", the 
latter being defined as ones in which the identity of each individual business 
is significant and the data collected determines the particular treatment of that 
business. 


A "business" is defined as a legal entity (or group of legal entities), excluding 
government departments or agencies, engaged in the production or sale of 
goods or services in Australia. Organisations such as private schools, 
hospitals, voluntary organisations, designated under the law as "not for profit" 
but producing or selling goods or services are regarded as businesses as 
they feel respondent burden in just the same way. 
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"Commonwealth Government departments and agencies" is interpreted to 
include organisations that are considered part of general government for the 
purpose of the National Accounts, excepting some that are not "budget 
funded" as defined in compilation of public and private financial statistics. 


Review Criteria 


The review criteria are the kernel of the clearance process. The Task Force 
report (Chapter 5) and the Cabinet Submission provided guidance regarding 
the nature of the review criteria, requiring the SCH to ensure that: 

- there is no adequate alternative source of information available and no 
reasonable, alternative means of obtaining the required information with less 
provider load; 

- the collection methodology is appropriate for the objectives of the 
collection and the intended uses to be made of the results, in particular, the 
frame provides adequate coverage, the collection forms have been 
appropriately tested, the expected level and quality of responses are justified, 
statistical standards are used where appropriate, and proposed processing 
and estimation systems are adequate; 

- a group of businesses or business associations have been consulted 
about the nature and objectives of the collection and data availability, and 
there is an assessment of provider load; and 

- there are adequate systems (both computer and people-based) to 
ensure the collection is conducted and processed in a manner that will 
provide output of quality appropriate for its intended uses. 


Within this general framework, operational review criteria have been 
established. They are based on various survey design guidelines, quality 
checklists, and information from similar programs at other agencies. An 
important consideration in their formulation was the intended depth of the 
review process. The more detailed the review, the more potential for finding 
ways in which survey can be substantially improved, but the more information 
about the survey has to be furnished by the survey manager and the longer 
the review takes. There is a practical balance to strike between thoroughness 
and practicality. Thus, the choice of review criteria has been influenced not 
only by respondent burden and quality considerations, but also by the 
average time that can be devoted to reviews (roughly estimated from the 
number of surveys to be reviewed and the resources available) and the 
information that survey managers and designers can reasonably be expected 
to have at their fingertips. 


The review criteria and the survey information required to support them have 
been incorporated in an electronic "information template", which is sent to 
each survey manager at the beginning of the clearance process. In the case 
of ABS collections, the template is already populated with information from an 
internal metadata system (the Collection Management System) so that the 
survey manager does not have to supply again information that has already 
been recorded there. 
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Since its initial introduction, the information template has been streamlined in 
response to survey manager feedback, mostly comprising complaints about 
the difficulties in understanding what was required and time spent in 
gathering together the information. The latest version of the template is 
appended. It does not contain any information that the manager or designer 
would not need to know in order to design and conduct the survey efficiently, 
effectively and with minimum respondent burden. 


One point regarding the criteria is worth highlighting. It is assumed that each 
Government agency is an authority regarding its own information 
requirements. These requirements are not subject to SCH questioning. 
However, they must be well documented, and the SCH does require evidence 
that a survey is necessary to meet them 


Clearance Procedures 


Development of clearance procedures began with identification, elaboration, 
and documentation of the core SCH functions, as outlined in the following 
paragraphs. A solid infrastructure of well defined procedures provides basis 
for ongoing continuous improvement. 


Procedures for identification of Commonwealth Government business 
surveys. Within the ABS this is a relatively simple process as, in principle, 
every survey collecting information intended for public dissemination is 
recorded in the "Collection Management System". All that had to be added to 
satisfy SCH needs were records for pilot tests, survey evaluation surveys, 
and other program and product evaluation surveys not previously recorded in 
the system. 


Identifying business surveys conducted by other agencies is not so 
straightforward as there is no equivalent list. (In fact, as previously noted, 
such a list will be a significant by-product of the clearance process.) Survey 
identification is a primary role of the survey liaison officers. Their efforts are 
being supplemented by periodic review of Government consultancy contract 
proposals submitted for approval, and ad hoc examination of Government 
internal bulletins, newsletters, etc. 


Survey review procedures. \|deally, every review starts with face to face 
contact between the SCH reviewer and the survey manger and/or designer. 
The key dates in the review timetable are identified and the mechanism - 
usually the Internet - for communicating information about the survey is 
established. The earlier the SCH becomes involved the more likely the 
review is to bring benefits rather than delays. However, there is also a 
balance to be struck here, as the SCH does not have the mandate or 
resources to comment in detail on proposals that are still at a very preliminary 
stage. 
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The time spent on a review is determined in accordance with its potential for 
improving the survey or reducing respondent burden. Factors taken into 
consideration are: 

- the size of the survey in terms of respondent burden - the larger the 
survey, the more scope for reduction in burden; 

- the organisations designing and conducting the survey - the less their 
experience, the more potential for identifying improvements; and 

- the type of survey according to its intended uses - the more 
widespread the use, the more widely beneficial improvements will be. 


In this last mentioned respect, surveys are classified into two basic types: 
"public interest" surveys and "other". Public interest surveys are those 
collecting data that are to be publicly and widely disseminated. Most ABS 
surveys fall within this category, as do many of the surveys conducted by 
organisations such as ABARE, Australian Institute of Health and Welfare, etc. 
The "other" category includes survey pilot tests, survey evaluation studies, 
and the whole raft of client satisfaction (program and product evaluation) 
surveys. This category of surveys receives less attention. In particular, for 
product evaluation surveys (of which ABS readership surveys are an 
example) whilst basic information, including respondent burden, is collected, 
there is no formal review or approval as such. This is because the likely 
benefits of reviews are outweighed by the effort required to provide the 
information for, and to conduct, the reviews. 


Approval and non-approval procedures. Surveys satisfying the review criteria 
receive an approval number to be displayed on all survey material that is 
seen by respondents. Survey managers have the option of using the SCH 
approval logo. As SCH activities become well known use of the logo may 
begin to improve response rates. 


At the same time as approval is given, the survey manager is asked to 
identify any information about the survey that should not be made publicly 
available via the Internet. It is not envisaged that there will be much 
information in this category at least not for ABS surveys as they are 
conducted for public good at taxpayers expense and the information can in 
any case be obtained under the Freedom of Information Act. 


A non-approval situation requires great sensitivity. A non-approval can be 
appealed by the agency head to the Australian Statistician. When a 
non-approval seems imminent, the ABS Division Head with responsibility for 
the SCH is briefed. Non approval is a last resort. A much more frequently 
occurring situation is where the SCH sets certain conditions for approval, and 
provides approval when these are met. 
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Procedures for post survey evaluation. After a survey has been conducted, 
the survey manager is asked to complete a post survey template and to 
confirm that the survey has been conducted according to plan or to document 
significant variations. 


Dissemination - Commonwealth Register of Surveys 


A by-product of the clearance process is the Commonwealth Register of 
Surveys containing information that has been provided to the SCH during the 
clearance process. Dissemination via the Internet essentially replaces the 
need for a single omnibus printed publication, which, based on past 
experience would be obsolete before it was produced. The Register will 
become an important element of the clearance process itself as, over time, it 
will be a primary source listing surveys that have already conducted and their 
objectives. This should greatly reduce the potential for duplication. 


Systems 


A phased approach to systems development has been adopted. The SCH 
began survey clearance in November using standard (Notes) software 
available with the ABS, and standard Internet features. As the systems 
requirements become more precisely identified through practical experience, 
the software is being enhanced and refined in stages, so that, ultimately, it 
will support direct update and access to review results by survey managers 
outside as well as inside the Bureau 


Experiences at Other Statistical Agencies 


One of the starting points in the development of SCH procedures was an 
investigation of programs with similar objectives administered by national 
statistical agencies abroad. The investigation proved to be a very fruitful 
source of ideas. Several key aspects of SCH originate from these ideas 

adapted to Australian circumstances. 
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In the UK, procedures for control of Government statistical surveys were 
introduced in March 1992 with the stated objectives of minimising the burden 
placed on businesses and abolishing, simplifying, or reducing the sample 
sizes of, surveys wherever possible. The procedures were subsequently 
modified and expanded in response to recommendations from the Business 
Deregulation Task Forces, and were elaborated in a letter from the Prime 
Minister to Ministerial Secretaries, August 1994. Their essential features are 
that: 

- new surveys and significant changes to existing regular surveys 
require prior ministerial authority; 

- all surveys conducted at regular intervals should be thoroughly 
reviewed by Ministers at least once every five years, and in the case of 
monthly or quarterly survey, every three years 

- proposals for new surveys and reviews of regular surveys must be 
cleared through the Survey Control Unit of the Office for National Statistics 
and include a rigorous assessment of compliance costs. 


The UK clearance model profoundly influenced SCH implementation in terms 
of the design of clearance procedures, their promulgation through the letter 
from the Australian Statistician, the identification of survey liaison officers 
within each department, and the allocation of clearance effort according size 
of survey . This does not imply slavish use of the UK model. There are some 
quite distinct differences reflecting the more centralised statistical system in 
Australia, the broader objectives of the clearance program, and the 
opportunities to use more modern information technology. In particular, the 
SCH review is to a considerably greater depth, mirroring the objective of 
improving output quality as well as reducing respondent burden. Also survey 
information is collected in electronic form and disseminated via the Internet in 
contrast to the UK's paper based approach. 


In France there is a parallel program for review of business surveys, though it 
is confined to surveys of public interest. The program which is run by the 
Institute Nationale de Statistiques et Etudes Economiques (INSEE) includes 
annual review of all economic surveys by the French National Council for 
Statistical Information, also the assignment of a "general interest label" to 
surveys that satisfy "conformity" criteria. 


The National Council was established in 1972 and expanded into its present 
form in 1984. Review by the National Council is at broad conceptual level. 

Its objectives are to ensure that each survey program meets a genuine 
information requirement in the public interest, that it is not redundant, and that 
it contributes to the medium term goals for the field under study. The role of 
the "Label Committee" was defined in 1994. Review by the Label Committee 
complements that of the National Council by focusing on survey methodology 
with the objective of ensuring appropriate survey practices are used. 
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In the US, the program for clearance of all Federal Government business 
surveys has been in place for at least a decade. The Office of Management 
and Budget (OMB) reviews business surveys in accordance with rules and 
regulations for controlling paperwork burdens on the public based on the 
Paperwork Reduction Act. Despite a critical audit of the program in the early 
1990s, it was reaffirmed in 1995. The review process includes both public 
interest and program/product evaluation surveys. Interestingly, the OMB has 
apparently been taken to court by a business for clearing a survey that the 
business asserted should not have received clearance. 


Until 1991, Statistics Canada used to have a program operating on similar 
lines to the UK ONS Survey Control Unit and the US OMB program, but 
including all surveys not just business surveys. It was eliminated in a round of 
budget cutting in 1991, being considered of less benefit than other parts of 
the statistical program. 


PART C: ORGANISATION 
Communications 


The major stakeholder types, internal and external, have been identified. The 
roles and appropriate communications arrangements for each type have been 
defined. Particular emphasis has been placed on the opportunities for 
partnerships, and for harmonising activities with other areas 


Prior to the Australian Statistician's October letter detailing SCH procedures, 
meetings were held with senior managers in five other Commonwealth 
Government agencies believed to have significant business survey activities. 
The objectives of the meetings were to outline the broad level SCH proposals 
and ensure their support, and to identify any issues to be taken into account 
in preparation of the SCH procedures. 


Every opportunity is being taken to publicise SCH objectives, procedures and 
accomplishments through face to face meetings, seminars and articles in 
Government publications and, most recently, the Commonwealth Register of 
Surveys. SCH information and training material and services are being 
developed in collaboration with the Statistical Consultancy and Training 
Section. 
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Survey Liaison Officers have been identified in all departmental portfolios and 
agencies with business survey activity. A SCH User Group (SCHUG) has 
been established and meets regularly. It has representatives from external 
and ABS survey areas, from the Office of Small Business, and from potential 
users of SCH information. Maximum use is also being made of existing links 
to agencies through ABS outposted officers and the through the Statistical 
Network. 


Reporting and Evaluation 


The ABS is required to provide an account of the SCH in its Annual Report. 

In addition, the Office for Small Business, in its role of following up on 
implementation of the Small Business Deregulation Task Force Report 
recommendations, is the primary SCH sponsor and requires progress reports 
on a regular basis. The SCH also reports to its User Group and an internal 
ABS Implementation Committee. Also, as previously noted, mid way through 
the second year of operation, there will be an evaluation of the SCH. The aim 
will be to complete the evaluation in time to incorporate its recommendations 
into plans for operation from July 1999 onwards 


PART D: OPERATIONS 
Review Schedule 


The framework within which the review schedule has been set is that, 
beginning November 1997: 

- all new or substantially redesigned surveys must be reviewed before 
going into the field; 

- the complete suite of regular ongoing surveys must be reviewed by 30 
June 1999; and 

- larger regular surveys (typically those with more than 2000 
respondents) should be reviewed before smaller ones and by 30 June 1998 if 
possible. 


Within this framework, allowance has been made for practical realities. For 
example, some large surveys are being redesigned and their reviews cannot 
begin until after June 1998. 


Review Results and Impact on Quality and Respondent Burden 


As of 1 May 1998, SCH staff had completed 17 reviews (8 ABS and 9 
external) and were actively engaged in a further 17 reviews (12 ABS and 5 
external.) Of the completed reviews, all but one had resulted in an approval. 
The disapproved survey was not conducted as had originally been proposed, 
the information sought being simplified to the point were the resulting survey 
was no longer subject to SCH review. 
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Although it is too early to be sure whether the benefits of the SCH justify the 
costs, reviews can produce positive outcomes in terms of quality or 
respondent burden, as evidenced by the following examples. 

- SCH concerns about the frame, response rates and questionnaire for 
an Internet Commerce Survey resulted in the sample being reduced from 
1100 to 100 and the mailback questionnaire replaced by a personal interview 
with much more focus on respondent's understanding of the questions. 

- Concerns about business respondents' capacity to provide the details 
requested in the proposed DWRSB Award Coverage Survey were validated 
by the pilot test and the survey was delayed to enable redesign of the 
questionnaire. In the absence of SCH intervention, the survey would have 
gone ahead without the results of the pilot. 

- The review of the ABS Business Expectations Survey identified two 
superfluous questions which have since been eliminated from the 
questionnaire. 


On average, eight days were spent on each of the 17 completed reviews. In 
five cases some quality improvements were made and in four cases there 
were reductions in respondent burden. In two reviews, the SCH was 
complimented on its work, but in 12 others there were complaints from survey 
managers. The commonly lodged complaint is about the volume and 
complexity of information that survey managers are obliged to supply. 
However, when pressed, those complaining cannot identify which information 
requirements or review criteria should be eliminated. Indeed, on more than 
one occasion, the information template and review criteria have been 
described as providing "a good checklist". 


PART E: CONCLUDING REMARKS 
Issues and Challenges 


The initial reaction of most survey managers is to see the SCH asa 
bureaucratic impediment, involving additional effort to furnish the information 
required for the review, the risk of jeopardising the survey timetable, and no 
tangible benefits. The challenge for SCH staff is to provide sufficient useful 
advice that the review is seen, with hindsight at least, as adding value. We 
emphasise that the SCH does not ask for any information that should not 
already be documented. Furthermore, in the case of surveys that are 
conducted by a consultant, we stress to the survey manager the potential 
benefits of the review as a check on the quality of the work that they are 
receiving. 
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In many cases, there is little time to conduct the review and identify changes 
suggested or required for the survey before the questionnaire is scheduled for 
printing, or, worse still, is due in the field. There are two basic reasons for 
this. First, survey clearance is relatively new and survey managers are not 
aware of the responsibility to put their surveys through the process, or of the 
lead time this requires. Second, and particularly in the case of program and 
product evaluations, the need for a survey and its design occur in a very short 
time frame. The challenge for the SCH is to promote its activities in such a 
way as to ensure that consideration is given to survey clearance right from 
the very start. In the case of surveys conducted by consultants, review and 
approval should be built into the contract. The sooner the SCH is involved, 
the more opportunity to identify major problems early and the less likelinood 
there is of a delay in schedule. 


Often, especially in the case of regular ongoing public interest surveys, the 
survey objectives are well articulated, but the underlying information 
requirements that a survey is designed to satisfy are poorly expressed. 
Although the SCH does not question information needs, it does demand that 
they are well documented as only by this means can alternative sources of 
information be properly assessed for potential duplication. 


Sample sizes are rarely well justified. Quite commonly, the sample size is set 
to the maximum that the survey budget can stand rather than the reverse. 
When the target population is relatively small, say 2000 businesses or less, 
there is a natural tendency for survey managers to prefer a census rather 
than a sample survey, even where small area or other detailed breakdowns 
are not required. The simplicity of a census is considered to outweigh its 
extra cost. The challenge for the SCH is to make departments and agencies 
more conscious of the respondent burden their surveys impose, and to be 
more inclined to factor it into sample size decisions. 


Particularly in the case of smaller, one off surveys, there is lack of concern 
about low response rates and there implications. SCH staff have 
encountered several instances where the survey manager has expressed a 
preference to conduct a census and expect a low response rate, say 40%, 
than to aim to get the same number of respondents from a, say 50%, sample. 
The SCH approach is to emphasise that the results cannot be interpreted as 
representative of the population unless the response rates are reasonable, or 
unless there is some other evidence that the respondents are representative 
of the non-respondents. Here "reasonable" depends upon the intended use 
of the results, but is likely to be at least 60%. 
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Finally, there are several instances where regular ongoing surveys with 
apparent duplication in their underlying information requirements and 
objectives should be reviewed as a group, but for practical reasons, e.g., 
differences in survey reference period, have to be scheduled for review 
separately. Specific examples are: 

- the ABS's agricultural commodity and agricultural finance surveys and 
ABARE's agricultural and grazing industries and diary farm surveys; 

- ABS surveys collecting employment related data items as their 
principal focus and other ABS surveys collecting employment data in 
conjunction with financial or commodity items; 

- surveys collecting tourism data. 


The approach that the SCH is adopting in these cases is to conduct the 
reviews separately but to give approval for a limited period only and to 
accompany it with comment to the affect that between now and the next 
approval the issue of overlap with other surveys in the group will be the 
subject further SCH scrutiny and will need to be addressed prior to the 
following approval. 


Where to from Here 


Visions for future development include the following. 

- In partnership with State and Territory governments, expanding the 
clearance process and/or register to embrace State and Territory surveys. 

- Ensuring that an entry is automatically generated in the Register of 
Surveys at the time that a new survey proposal is first put forward for funding. 
- Expanding the Register to include descriptions of administrative 
processes that generate statistics and of other surveys that are not covered 
by the clearance process but are of public interest. Under this scenario the 
"Register" would become a "directory". 

- Including links from the Register (directory) to the data outputs 
produced by the surveys. This would greatly enhance the value of both the 
Register and the data products. 


Ultimately the goal would be to educate Government survey managers and 
designers to the point where response burden and survey quality 
consciousness were sufficiently well embedded that the SCH review could be 
scaled back to an audit role, or disappear entirely, leaving in place a register 
in which all survey descriptions were automatically recorded. This would be 
akin to building quality and documentation right into survey taking processes 
rather than carrying out quality control and monitoring after the event. 


APPENDIX 
STATISTICAL CLEARING HOUSE INFORMATION TEMPLATE 
Containing Review Questions and Criteria 
(Revised February, 1998) 


Information provided in answer to the questions below, labelled Al, A2,....B1, B2, etc., 
forms the basis for the Statistical Clearing House (SCH) review of a survey. 


The questions are grouped into sections. Survey managers are requested to provide 
answers to each section as soon as the relevant information becomes available 


Within each section, the corresponding review criteria are labelled SCHO1, SCHO2, etc. 
SCH staff review a survey in relation to these criteria, based on the answers provided by 


survey managers to the questions. There are no review criteria for Section A . 


For a repeating survey, the questions refer to the first survey cycle that is subject to 
clearance. 


A. General Information 


Al. What is the name of the survey? 


_—_—_—_—————_——s—=é_—— 


A2. Which agency and area is responsible for the survey? 


Note 
Area refers to section, branch, division or equivalent in your agency. 


PT 


A3. Who is the survey manager and/or principal contact person for survey 
clearance? 


Include 
Phone number and email address (where applicable). 


PT 


A4. Is the survey voluntary or mandatory? If mandatory, under what 
authority? 


PO 


A5. What agency (or agencies) is (are) funding the survey? 


[| 


A6. Is the survey a new survey or a cycle of an existing repeated survey? 


| 


A7. To what reference period(s) does the survey refer? 


Note 
Reference period refers to the period of time for which data are collected and/or compiled (eg 
calendar year, financial year, specific day of month, etc). 


pe 


A8. Is the survey to be conducted once only or repeated? 


PO 


(For repeating surveys only) 


A9. With what frequency is the survey to be repeated? 


PO 


A10. Are there likely to be significant changes between survey cycles, and if so 
what are their general nature? 


Note 

Factors that may cause a significant change are: 

. changes in the principal outputs such as a change in classifications or data items used, or an 
increase in the number of data items; 

. a change in the data collection methodology (eg from self-completion to telephone 
interviewing); 

. achange in the sample design; 

. a change in the frequency of data collection. 


PO 


B. User Requirements / Survey Objectives / Budget 


Bl. Why do you need to conduct a survey? 


Note 

. Refers to the underlying information requirements that the survey is intended to address. 
For example, what problems are you trying to solve? What decisions need to be made? 
What specific policies or actions are being determined or assessed? 

. Please include who requires the information. 


PT 


B2. How will the survey meet this need? That is, what are the objectives of the 
survey in terms of content and constraints? 


Note 
. Under 'content' please list what topics the survey will cover. 
. Please include any of the following constraints if applicable: 
- fixed costs 
- fixed sample size 
- fixed accuracy (eg target RSEs) 
- any other constraints 


PO 


B3. What is the timetable for the survey? 


Include 

. testing 

. the date data collection commences 
. the date data collection finishes 

. when results are available 


Pd 


B4. What is the budget for developing and conducting the survey? 


Include 
A breakdown of the budget, eg fieldwork, testing, coding, data processing. 


| 


Review Criteria: 


SCHO1 Have the information requirements that the survey is to address been 
adequately described? 


SCHO02__Do the objectives of the survey match the underlying information 
requirements? 


SCH03 Do the objectives appear to be achievable within the budget? 


SCH04 Does the survey timetable satisfy the underlying information requirements 
and does it appear to be achievable? 


Cc. Alternative Sources of Data 


C1. What alternative sources of data have been considered? 


Include 
. surveys during the previous two years that collect similar data 
. other sources apart from surveys 


PT 


C2. In what respects are these alternative sources insufficient? 


| 


C3. What consideration has been given to working with other agencies to make 
their data more suited to your needs? 


PT 


Review Criteria: 
SCHO5 Have alternative sources of data been investigated? 


SCH06 _ Is full use being made of other sources of data? 


D. Data Outputs 


For repeating surveys, please provide copy (copies) of previous publication(s). 


D1. What are the principal data outputs to be produced? 


Note 

. Refers to the main data items to be produced by the survey and how they will be 
cross-classified (eg by geography, industry, size, sector of business, etc). 

. Copies of previous publications, final reports or outlines of proposed reports can be 
provided to help answer this question. 


PO 


D2. What consideration has been given to the use of standards? Please specify. 


Include 
. Standard data items such as employment, housing, etc. 
. Standard classifications such as industry, geography, commodity, etc. 


PO 


D3. What consideration has been given to making data items consistent between 
survey cycles or across surveys? 


_ 


Review Criteria: 
SCH07 _ Do the principal data outputs satisfy the survey objectives? 
SCHO08 — Have standards been used where appropriate? 


SCHO09 Has consideration been given to the consistency of data items between 
survey cycles or across surveys to facilitate comparison or joint use? 


E. Questionnaire Design and Data Collection 


Provide copy (copies) of questionnaire(s). 


El. Are there any questions not used directly in the principal data outputs 
specified in D1? If yes, how are these questions used? 


fe 


E2. What is (are) the method(s) of collecting the data (eg self-completion, 
telephone interviewing, face to face interviewing, etc)? Why was it (were they) 
chosen? 


Note 
If more than one method, please indicate the extent to which you expect each method to be 
used, eg 5% of responses by fax and the remaining responses by phone. 


a) 


K3. If only one method is stated in E2 above, what consideration has been given 
to offering respondents a choice in how they provide their answers (eg phone, 
fax, mail)? 


PT 


E4. What testing of the questionnaire(s) has (have) taken place? 


PT 


E5. What other consultations have taken place with businesses or business 
associations regarding availability of data items and data collection methods? 


Note 
Refers to activities other than testing. 


PT 


E6. What training of staff involved in data collection has taken place or is 
planned? 


Include 

. Interviewers 

. Staff involved in follow-up activities 

. Staff involved in answering queries from respondents 


PO 


Review Criteria: 

SCH10 Do the questions to be asked match the statistical tables to be produced? 
Are there superfluous questions, ie does every question have a 
justification? Are there missing questions? 

SCH11 Has the choice in data collection method(s) been justified? 


SCH12 Do the data collection methods involve minimum respondent burden? 


SCH13 Are the data collection methods appropriate for the data items to be 
collected? 


SCH14 — Has (have) the questionnaire(s) been designed and tested adequately? Is 
there evidence that all questions can be answered? 


SCH15 Have staff been effectively trained to minimise non-sample error? 


F. Respondent Burden 


F1. On average how long will a respondent be expected to spend in completing 
each questionnaire? How has this estimate been derived? 


PO 


F2. What steps have been taken to reduce the burden on small businesses, eg use 
of a reduced questionnaire? 


PT 


F3. What commitments have been made to preserve the confidentiality of 
respondents? How will these be implemented? 


Ee 


Review Criteria: 
SCH16 Are estimates of respondent burden credible? 


SCH17 Does the estimated respondent burden imposed by the survey appear 
reasonable given the survey objectives? 


SCH18 = Have confidentiality concerns been appropriately addressed? 


G. Population / Frame 


G1. What is the target population for the survey? 


Note 

. The target population, or scope of the survey, is the population about which information is 
sought. 

Include 

. any geographical, industrial and size constraints 


PT 


G2. How is the frame for the survey to be obtained? 


Note 
. The frame, or sampling frame, is the list of businesses from which the sample is selected. 
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G3. What is the type of unit on the frame to be used for the survey? 


Note 
. examples of type of unit are: location, establishment, management unit, enterprise, 
employer 
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G4. How many units are there on the frame? 


Note 
. this refers to the number of units, not the number of types of unit 


PT 


G5. What information on the frame about these units will be used for the 
survey? 


Note 
. examples of information which might be used are employment size, industry classification, 
whether the unit is in private or public sector, etc 
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G6. In what ways does the coverage provided by the frame differ from the target 
population? 


Include 
. an estimate of the proportion of units expected to be defunct (ie. no longer in business) 
. an estimate of the proportion of units expected to be out of scope 


———————— 


(For repeating surveys only) 


G7. How will the frame be updated for future survey cycles? 


PO 


Review Criteria: 


SCH19 Is the sampling frame appropriate given the underlying information 
requirements? 


SCH20 (For repeating surveys only) Will the sampling frame be adequately 
maintained? 


H. Sample Design 


H1. What sample design will be used, eg stratified simple random sample? 


Include 

. the sample selection strategy 

. if the sample design is multi-stage or multi-phase (or both), the sample selection strategy 
used in each stage/phase 


PT 


H2. What is the total sample size and how has it been set? 


Include 

. any constraints which have been imposed (eg cost) 

. target relative standard error (RSE) levels on selected data output items (if any) and reasons 
for choice 


el 


H3. What stratification has been used in the sample design? 


Include 

. Stratification variables 

. stratification boundaries (noting any strata which are completely enumerated) 
. reasons stratification strategy adopted 

. if multi-stage/phase sampling is being used, report for each stage/phase 


PT 


H4. How has the sample been allocated to strata? 


Include 

. allocation formulae (if used), and from where data referenced by formulae obtained 
. minimum and maximum stratum sample size constraints 

. reasons why the allocation strategy was adopted 

. if multi-stage/phase sampling is being used, report for each stage/phase 


Ea 


H5. What consideration has been given to ensuring respondent burden is spread 
evenly by controlling sample selection across surveys, or (for repeating surveys) 
over survey cycles? 


(Ea) 


H6. Of the total sample size reported in H2, how many units are expected to 
respond, and how many units are expected to be defunct (ie. no longer in 
business) or out of scope? 


Include 

. how these estimates were determined 

. if the sample is multi-phase then report these quantities for each phase 

. if quota sampling is used, please indicate the expected number of units to be approached to 
achieve the desired sample size 


PO 


H7. What allowances have been made for known frame deficiencies in the 
sample design? 


Include 
. allowances made for expected defunct units (ie. units no longer in business) 
. allowances made for expected out of scope units 


PT 


H8. What allowances have been made for expected non-response in the sample 
design? 


Include 
. allowances made for refusals and non-contacts 


PT 


H9. What evidence is there that the expected respondents will be representative 
of non-respondents? 


rs: 


Review Criteria: 
SCH21 Is the sample design appropriate given the survey objectives? 


SCH22 Are the expected response rates realistic, given survey authority, data items 
and collection methods? 


SCH23 _ Is the expected non-response bias tolerable given the survey objectives? 
SCH24 _ Is there scope for a reduction of small businesses in the sample? 
SCH25 _ Is there scope for spreading respondent burden more evenly by controlling 


sample selection across surveys, or (for repeating surveys) over survey 
cycles? 


I. Data Processing, Estimation, and Analysis 


I1. What steps will be taken to ensure the expected response rate is achieved? 


Include 
. what follow-up procedures are planned 


Pe ed 


12. What quality control procedures will there be for data entry and coding? 


Note 
. for example clerical scrutiny of forms, on-line edits, etc. 
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13. Will outliers be identified, and if so how will they be handled? 


PT 


14. What sources of non-sampling error could have a significant impact on 
survey results and what is being done about them? 


Include 

. what imputation procedures will be used 

. what known frame deficiencies are to be treated in estimation 

. what other allowances will be made for other potential sources of non-sampling error in 
estimation 


PT 


15. Will data be aggregated into statistical tables, and if so what are the 
estimation formulae for the principal output data items? 


Include 

. how adjustments for outliers, imputation and frame undercoverage factors have been 
incorporated into estimation formulae 

. variance formulae for variances to be published 
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16. How are the results of the survey to be analysed? 


Note 
. relate response to answers provided in questions B1 and B2 


PO 


17. How are the results of the survey to be made available to the major users? 
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18. What processing performance measures have been defined? 


Note 

. This question is concerned with processes which are in place to ensure that the timetable 
deadlines will be met, eg responses received by date, percentage of data captured by date, 
percentage of data edited by date, etc. 


PO 


19. How have the data processing, aggregation and dissemination systems been 
tested? 


—————EEEEEEE 


Review Criteria: 


SCH26 _ Is the quality control of data processing procedures adequate given the 
underlying information requirements? 


SCH27 _ Is there evidence that the resulting data will be successfully processed, 
aggregated and disseminated? 


Post Survey Information 


Respondents 


1. Did the average time(s) actually taken by respondents to complete the 
questionnaire(s) differ significantly from what was planned? If so, what were the 
amounts and reasons for the differences? 


—— 


Original information (from F1): 


PO 


2. How many written complaints were received from respondents and what steps 
were taken to handle these complaints? 


PT 


Processing 


3. Did the achieved response rates differ significantly from what was planned? 
If so, what were they and why did they differ? 


PT 


Original information (from H6): 


ee 


4. Did the actual data entry, coding, editing and imputation procedures differ 
significantly from what was planned? If so, in what way? 


PT 


Original information (from I2 & 14): 
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Survey Results 


5. Did the broad level data outputs differ significantly from what was planned? 
If so, in what way? 


PT 


Original information (from B2): 
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6. Did the accuracy of the results differ significantly from the target accuracy? 
If so, what are the differences and why did they occur? 


PT 


Original information (from H2): 


PT 


7. Will the outputs (or some of the outputs) be made available to the public? If 
so, to which outputs (eg final report, selected tables, etc) should the 
Commonwealth Register of Surveys provide links? 


PT 


Costs 


8. Did the cost of developing and conducting the survey differ significantly from 
what was budgeted? If so, what was the cost and what were the reasons for the 
differences? 


| 


Original information (from B4): 


PT 


